🕷️️ Job Radar • SCRAPING

freelancer.com 2026-04-14 🟡

🔹 Consistent Data Extraction from Australian Automotive Website
👤 Client: 🇦🇺 Hamilton hill, Australia Member since 2025-06-11
💰 Price: $573 Average bid
🚩 Problem: Extract comprehensive vehicle data from a list of URLs to ensure accurate and consistent market-wide comparisons.
📦 Existing: List of 1500 URLs, each covering top 100 makes, models, grades, and variants in Australia. No existing datasets or tech stack mentioned.

Specifications:

[Target] Extract data from 1500 URLs
[Method] Use Python with Scrapy, BeautifulSoup, Selenium for web scraping; AI-assisted extraction if needed
[UI/UX] Not applicable
[Stack] Python, Scrapy, BeautifulSoup, Selenium, Pandas, Openpyxl
[Security] Ensure data privacy and compliance with website terms of service
[Format] Deliver sample Excel file (5-10 vehicles) for sign-off; final Excel workbook with one row per variant

Workflow:

1. Analyze the structure of a few URLs to identify common data points and their locations.
2. Develop a scraping script using Scrapy or BeautifulSoup, incorporating Selenium if dynamic content is involved.
3. Implement AI-assisted extraction techniques for complex or missing fields.
4. Standardize extracted data: ensure consistent column names, unit normalization, and categorical value standardization.
5. Test the script on a sample of 5-10 vehicles to confirm structure and accuracy before full-scale scraping.
6. Execute the full scraping process for all 1500 URLs.
7. Clean and normalize the scraped data using Pandas or similar tools.
8. Export cleaned data into an Excel workbook with clearly labeled columns.
9. Deliver sample Excel file for client approval.
10. Finalize and deliver the complete dataset.